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Abstract 

Background: Rapid developments in intensive care medicine have made mechanical ventilation an essential 
method in the resuscitation and comprehensive treatment of critical care patients. This study aimed to develop and 
evaluate an appraisal form assessing the clinical effectiveness of adult invasive mechanical ventilation systems. 

Methods: An appraisal form was designed according to the effectiveness evaluation theory of the American 
Weapons Systems Effectiveness Industry Advisory Committee (WSEIAC) along with literature review and expert 
panel review. Content validity of the preliminary form was analyzed in a cohort of 200 patients. Exploratory and 
confirmatory factor analysis was used to assess appraisal form validity. Discriminate validity of different ventilation 
outcomes was analyzed by f test. Test/retest reliability and inter-scorer reliability were evaluated with 30 patients 
after a 2-week interval by Cronbach's alpha. 

Results: Exploratory factor analysis showed eigenvalues for 3 dimensions (availability, dependability, capability) to 
be 7.85, 4.43, and 4.22, respectively. Cronbach's a for internal consistency of the appraisal form was 0.957, and 0.922, 
0.961 and 0.937, respectively, for the 3 dimensions. Test-retest reliability of 3 dimensions was 0.976, and 0.862, 0.857, 
0.885, respectively. Intra-class correlation coefficient verified test-retest reliability; ICC 0.976 and 0.862, 0.857, 0.885 for 
3 dimensions, respectively. 

Conclusions: The appraisal form for clinical effectiveness of adult invasive mechanical ventilation systems has high 
reliability and validity and may be used in clinical setting. 
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Introduction 

Rapid developments in intensive care medicine have made 
mechanical ventilation an essential method in the resusci- 
tation and comprehensive treatment of critical care 
patients [1]. About 80% of patients in intensive care units 
are reported to require mechanical ventilation, and nurs- 
ing care of patients receiving mechanical ventilation has 
become increasingly important, including nurse-led wean- 
ing of ventilator patients [2]. While hospitals in other 
countries may rely on trained respiratory care therapists 
to assess and care for mechanical ventilation systems [3], 
in the intensive care units of China, nursing staff especially 
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trained in basic knowledge and techniques of intensive 
care medicine provide the main care for critically ill 
patients, including airway care for patients requiring 
mechanical ventilation. An important role of clinical nurs- 
ing staff is to correctly assess the effectiveness of mechan- 
ical ventilation systems during respiratory intensive care 
in order to guarantee safe and effective treatment of 
patients [3-5]. While mechanical ventilation potentially 
saves patients' lives, the complexity and diversity of 
tracheostomy tubes, specific care of surgical sites and 
other nursing issues such as unplanned intubation and 
complications may have an impact on its clinical effective- 
ness [5]. In North American hospitals, while care of the 
mechanical ventilation system may be in the hands of 
trained respiratory therapist, airway assessment and man- 
agement remains a primary responsibility of nurses caring 
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for critically ill adults [5], Serious concerns of clinical nurs- 
ing staff and respiratory therapists about mechanical ven- 
tilation have led to investigations of the relative 
effectiveness of ventilation machines, endotracheal tubes 
and tracheostomy tubes [3-5], as well as the critical care 
environment [6,7]; personnel issues [8,9]; management of 
artificial airways [10,11]; and prevention of complications 
[12-14]. However, current research about nursing evalu- 
ation of mechanical ventilation mainly focuses on analyz- 
ing and discussing single influential factors, while studies 
of comprehensive nursing interventions for mechanical 
ventilation systems are lacking. This may be due in part to 
the fact that nursing staff may not be responsible for 
mechanical ventilation systems in some countries, while 
in China, ICU nurses are solely responsible for mechanical 
ventilation systems and related patient care. Therefore, the 
design of an appraisal form to evaluate the comprehensive 
clinical effectiveness of adult mechanical ventilation sys- 
tems may provide a valuable assessment tool for critical 
care nurses responsible for mechanical ventilation admi- 
nistered to patients in intensive care units in China. 

Mechanical ventilation systems comprise positive air- 
way pressure ventilation machines connected to patients 
via artificial airways {i.e., endotracheal tubes or tracheot- 
omy) [15]. The Joint Commission (TJC) (formerly the 
Joint Commission on Accreditation of Healthcare Orga- 
nizations [JCAHO]), is a US-based hospital accreditation 
organization that demands regular evaluation of the ef- 
fectiveness of mechanical ventilation systems along with 
other equipment used for patient care or diagnostic test- 
ing; TJC also requires that regulations and operating 
procedures must be maintained in written form for pro- 
fessional staff [16]. System effectiveness can be measured 
by any of several models and one of the most accepted 
is the effectiveness model advanced by the American 
Weapons Systems Effectiveness Industry Advisory Com- 
mittee (WSEIAC) [17]. The WSEIAC model has been 
used successfully to evaluate the effectiveness of various 
systems, including radar jamming [18], weapons systems 
[19], reconnaissance satellites [20], and preventive main- 
tenance of equipment [21]. According to the theory of ef- 
fectiveness evaluation of the WSEIAC model, there are 
three dimensions that define system effectiveness [17]. 
The first dimension of system effectiveness is availability. 
For mechanical ventilation systems, this means ensuring 
proper mechanical ventilation by conducting regular sys- 
tem checks, including testing of machine function, set- 
ting up proper alarms, maintaining the machine circuit 
and humidification device and selecting proper endo- 
tracheal or tracheotomy tubes [3-5]. Personnel training is 
also necessary to ensure that all nursing staff members 
or respiratory therapists possess the essential related 
knowledge and techniques for machine operation and 
patient care [21-24]. In addition, the working environment 



itself can have an impact on nursing quality and the mor- 
tality of intensive care patients [1,6,7]. Dependability is the 
second dimension of system effectiveness in the WSEIAC 
model, which includes maintaining patency of the airway 
as an essential nursing intervention [10,11,25,26], securing 
endotracheal and tracheotomy tubes in a safe position 
without pressure on the skin [5], reducing intratracheal 
damage by maintaining the lowest pressure of the endo- 
tracheal balloon and preventing air leaks by frequent 
checking and adjusting as needed and avoiding twisting or 
pulling on the endotracheal or tracheostomy tubes [27,28] . 
The airway also must be maintained when patients are 
being transferred or their position changed [11,29,30]. 
Patients under mechanical ventilation may also develop 
physical and mental reactions such as anxiety, fear, pain, 
sleeping disorders, depression and hallucinations [31,32], 
requiring that every patient with mechanical ventilation is 
assessed by nurses to establish an individual nurse-patient 
communication method and plan [15]. The third dimension 
of system effectiveness is capability, which is determined 
by the assessment of complications {e.g., ventilator- 
associated pneumonia and airway damage), unplanned 
extubation [33], aspiration [34,35] and patients' physical 
and mental reactions [36,37]. 

Based on the WSEIAC model of system effectiveness, 
this research aimed to develop and evaluate an effective- 
ness appraisal form for assessing adult invasive mechan- 
ical ventilation systems (EAP-AIMVS), and to apply it 
for measuring the effectiveness of nursing staff in estab- 
lishing artificial airways and helping to wean patients 
from mechanical ventilation. To our knowledge, this is 
the first system effectiveness appraisal form to be devel- 
oped for assessing mechanical ventilation systems based 
on the WSEIAC model. 

Methods 

We designed a cross-sectional study to develop and test 
an appraisal form for evaluating the effectiveness of 
adult mechanical ventilation systems. This study is 
divided into two stages: development of the appraisal 
form and an evaluation stage, including clinical valid- 
ation of the appraisal form. The Internal Review Board 
of the Nursing School of Fujian Medical University 
reviewed the study protocol and approved the study. 

Stage 1: Development stage 
Items development 

The effectiveness appraisal form for mechanical ventila- 
tion, including 3 dimensions and 45 items, was formu- 
lated based on the WSEIAC model [17], integrated 
literature review to formulate the item pool, and expert 
review by a panel of five clinical nurses. Literature re- 
view was conducted by the research team; previously 
published reports regarding scale development and 
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establishing mechanical ventilation system evaluation in- 
dexes helped to guide development of the items of the 
appraisal form [38-46]. The three dimensions of the scale 
are described as follows: (1) The first dimension is sys- 
tem availability (A), representing the state of the mech- 
anical ventilation system availability at all times (13 
items). (2) The second dimension is system dependabil- 
ity (D), representing the correct operation of the mech- 
anical ventilation by nursing staff (19 items). (3) The 
third dimension is system capability (C), representing 
the ability of the mechanical ventilation system to 
achieve goals of respiratory support (13 items). There- 
fore, system effectiveness (E) is defined as the compre- 
hensive effectiveness of mechanical ventilation system to 
meet the mission of ventilation within a given time and 
under intensive care environment. 

Content validity test 

Content validity of the appraisal form was tested by con- 
tent validity index (CVI), and evaluated by experts as 
previously described [39]. The panel of experts had 25 
members, including 13 experts in clinical intensive care 
medicine or research and 12 registered nurses, each with 
more than 10 years of working experience. Four scores 
were used to evaluate the appraisal form items: 1 = ir- 
relevant, 2 = some relevance, 3 = relevant, 4 = very rele- 
vant. Experts assessed every item and the relevance and 
feasibility of its dimensions, and advanced suggestion 
and opinions. The appraisal form was modified based on 
review and suggestions of the experts after deleting six 
items and integrating six items into other items, a pre- 
liminary appraisal form has 33 items in total. CVI of the 
appraisal form was 0.87 on average, indicating that every 
item reflects the theme concept of mechanical ventila- 
tion system effectiveness evaluation and can be studied 
further as previously described [40]. 

Preliminary examination 

To determine readability and practicability of the ap- 
praisal form, 30 patients were assessed by three clinical 
nurses using the appraisal form. Two items were deleted 
and three items only partially understood as expressed 
were revised. As a result, the final EAP-AIMVS had 31 
items deemed appropriate for further testing and ana- 
lysis in the clinical setting. 

Stage 2: Clinical validation of the appraisal form 
Study subjects 

The sample size for multiple factors analysis research 
should be more than 200 or 10 times the number of 
items according to a previously established formula [41]. 
Sample size for this study was established at 200 accord- 
ing to the number of items, progress of this study and 
feasibility. Patients receiving mechanical ventilation were 



selected by convenience sampling and 210 ICU patients 
were enrolled. Inclusion criteria were: patients age 
>18 years with invasive artificial airway (orotracheal or 
nasotracheal intubation, or tracheotomy.) and assisted 
mechanical ventilation; conscious patients willing to par- 
ticipate in this study, or unconscious patients with per- 
mission of family members. Exclusion criteria were: 
patients with noninvasive mechanical ventilation, or un- 
willing to participate in this study; patients without im- 
portant indicators and patients transferred to other 
hospitals during ventilation. Ten intensive care units in 
8 hospital in Fujian were selected as study locations. In- 
clusion criteria for ICU nursing staff were: permanent 
staff with nursing practice qualification certification and 
having had training for basic concepts, knowledge and 
techniques of ICU medicine. Nurses who were rotating 
ICU nursing staff and not permanent staff were 
excluded. The data collection period was from September 
2010 to March 2011. Signed informed consent was re- 
ceived from all participants or their responsible family 
members. 

Assessment tool 

Demographic data: We recorded gender, age, APACHE 
II score, type of ventilation machine and artificial airway, 
time of mechanical ventilation and successful weaning of 
each patient from mechanical ventilation. 

Assessment form scoring: The effectiveness appraisal 
form for assessing adult invasive mechanical ventilation 
systems is a nurse-administered assessment tool with 
three dimensions and 31 items. (Additional file 1: Table SI: 
Original EAP-AIMVS questionnaire) Four grades were 
recorded for each item: fully achieved, mostly achieved, 
hardly achieved and not at all achieved, and scores were 
4, 3, 2, and 1, respectively. The higher scores are, the 
higher the level of clinical effectiveness will be. 

Data collection method 

Before collecting data, the effectiveness appraisal form 
for adult invasive mechanical ventilation system was 
audited and approved by the hospital internal review 
committee. Researchers then used the effectiveness ap- 
praisal form to assess patients with nonparticipant ob- 
servation, meaning that except for researchers and 
coordinators of each study location, nursing staff were 
not aware of researchers' identity. Researchers observed 
every index of the mechanical ventilation system as trai- 
nees, collecting data by referring to medical records, and 
assessed every item of the appraisal form. A total of 210 
appraisal forms were filled out, and 210 appraisal forms 
were returned. Return rate was 100%, and 200 appraisal 
form were effective, attaining an effectiveness rate of 
95.2%. 
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Data analysis 

All statistical analyses were performed using SPSS 15.0 
for Windows statistics software (SPSS Inc, Chicago, IL, 
USA). Subjects' demographics and characteristics were 
summarized as mean with range (min. to max.) for con- 
tinuous variables and n (%) for categorical variables. 
Demography statistics used descriptive analysis (fre- 
quency, mean, standard deviation and median). Statis- 
tical significance was established as P < 0.05. 

Item analysis 

Item analysis assesses relativity between each item and 
total scores. When the correlation coefficient of each as- 
sessment item and total scores was more than 0.9, it was 
considered redundant. When less than 0.4, the item was 
deleted as not reflecting the concept to be assessed. 

Construct validation 

Construct validity of the appraisal form was assessed by 
exploratory factor analysis (EFA) and confirmatory factor 
analysis (CFA). Exploratory factor analysis evaluates the 
structure of factors and the consistency between each 
item and factor structure. Kaiser-Meyer-Olkin Measure 
of Sampling Adequacy and Bartlett's Test of Sphericity 
were used to judge whether factor analysis can be 
employed. Principle Component Methods was employed 
to select varimax orthogonal rotation. During factor ana- 
lysis, guidelines for filtration of assessed items were: (1) 
each factor's eigenvalue >lj (2) loading capacity of items 
with related factor >0.4; (3) each factor at least covers 3 
items; (4) tested by scree plot. 

To validate matching level of sample data and factor 
structure of the appraisal form, linear structural equa- 
tion modeling software [AMOS 17.0, Chicago, SPSS Inc] 
was employed for confirmatory factor analysis of the ap- 
praisal form items that were attained by exploratory fac- 
tor analysis. Fitting degree was evaluated with j^/di, 
comparative fit index (CFI), normed fit index (NFI), root 
mean square error of approximation (RMSEA). Model 
fitting effectiveness is acceptable when j/ 2 /df < 3, CFI 
> 0.9, NFI > 0.9, RMSEA < 0.08, according to a previous 
report [43]. 

Congruent validity 

Congruent validity was tested by comparing scores of 
different groups to see whether there were significant 
differences, as described in a previous report [44]. This 
study divided mechanical ventilation patients into a 
'weaned from ventilator successfully' group and a 
'deceased died during ventilation' group, and compared 
the scores of these two groups to determine differences, 
using discriminate analysis to evaluate the descriptive 
validity. Results were shown for Wilk's Lambda test and, 
and respective sensitivity, specificity. 



Reliability testing 

Internal consistency reliability, test-retest reliability and 
inter-scorer reliability were selected to evaluate the ap- 
praisal form. Cronbach's alpha was calculated to analyze 
internal consistency of each dimension and the form as 
a whole. Thirty patients were randomly selected within 
an interval of 2 weeks to evaluate test-retest reliability 
by calculating weight coincidence coefficients of these 
two assessments. Since the appraisal form is a nurse- 
administered evaluation form, intraclass correlation coeffi- 
cients were done by 2 scorers evaluating the same 30 
patients in the same period to test inter-scorer reliability. 

Ethical considerations 

This research was approved by the thesis defense com- 
mittee for masters degree and by the internal review 
boards of 8 hospitals in Fujian province whose ICUs 
were included in the study. Before data collection, con- 
scious patients and family members of unconscious 
patients were fully informed of the purpose and method 
of this study, and all provided signed informed consent. 
All appraisal forms were completed anonymously. 

Results 

A total of 200 subjects (142 males/58 females) were en- 
rolled in this study. The mean age was 58.3 years 
(Range: 16 to 99), with 42 subjects younger than 45 years 
(yrs), 35 aged between 45 to 55 year, 46 aged between 55 
to 65 yrs, 38 aged between 65 to 75 yrs, and 39 older 
than 75 yrs. The mean time in duration of mechanical 
ventilation was 216.8 hours (range: 2 to 2880 hours). 
Ventilation outcomes for the 200 subjects included 123 
(61.5%) weaning patients and 77 (38.5%) deceased 
patients during ventilation. (Table 1) 

Items analysis 

The EAP-AIMVS analysis result for 31 items was: 9 
items deleted whose relativity with total score was <0.3, 
and 22 items were reserved for construct validity test. 

Construct validity 

The exploratory factor analysis (EFA) used the principle 
component method for selection and performing varimax 
rotation for the 200 subjects. EFA results are summarized 
in Table 2. The KMO index = 0.946 and f = 4359 from the 
Bartlett's Test of Sphericity (P < 0.01) indicated that this 
dataset could be applied into EFA. (Results are not shown 
in tables.) Three factors were selected according to the cri- 
teria of Eigenvalue > 1, the three observed eigenvalues 
greater than 1 were 7.85, 4.43, and 4.22, respectively (with 
green background). The 22 items were classified into three 
factors representing the three dimensions of system as- 
sessment, as follows: availability (factor 1), dependability 
(factor 2), and capability (factor 3). Factor loadings were 
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Table 1 Subjects' demographics and characteristics 



Variables (N = 200) 

Age 58.3 (16 to 99) 

<45yrs 42(21.0) 

45 - 55 yrs 35(17.5) 

55 - 65 yrs 46 (23.0) 

65 - 75 yrs 38(19.0) 

g 75 yrs 39(19.5) 

Sex 

Male 142(71) 

Female 58 (29) 
APACHE II score 

<10 50 (25.0) 

11-20 67 (33.5) 

21-30 41 (20.5) 

>30 42(21.0) 
Ventilator type 

Servos 42(21.0) 

PB840 60 (30.0) 

Vela 36(18.0) 

Drager 62(31.0) 
Artificial airway 

endotracheal tube by mouth 123 (61.5) 

endotracheal tube by nose 30(15.0) 

tracheotomy 16(8.0) 

endotracheal tube by mouth + Tracheotomy 31 (15.5) 

Duration of mechanical ventilation, hour 216.8 (2 to 2880) 
Ventilation outcome 

Weaning 123(61.5) 

Died during ventilation 77 (38.5) 



Data were summarized as mean (Range: Min. to Max.) for continuous variables 
and n (%) for categorical variables. 

observed from 0.74 to 0.87 in factor 1, from 0.43 to 0.85 
in factor 2, and from 0.76 to 0.87 in factor 3. The cumula- 
tive variance explained by factors were 75.01%, including 
35.68% for factor 1, 20.15% for factor 2, and 19.18% for 
factor 3 (Table 2). 

Based on the constructed model, the CFA shows the 
final modified model yielded Chi-square = 34.79 (p < .001), 
df=13,//df=2.68, GFI = 0.98, AGFI = 0.94, NFI = 0.95, 
PNFI = 0.44, PGFI = 0.35, RMSE = 0.07 and exhibited 
good fit indices. (Figure 1) 

Congruent validity 

Regarding ventilation outcomes, 123 subjects (61.5%) were 
weaning and 77 (38.5%) died during mechanical ventila- 
tion, and the total EAP-AIMVS score was observed as 
80.41 (SD = 6.29) for weaning subjects and 63.13 (SD = 



10.19) for those who died during ventilation. (Table 3) 
The Wilk's Lambda test from discriminate analysis shows 
significant discriminate analysis (Wilk's Lambda = 0.432, 
P<0.05), with sensitivity and specificity equal to 91.1% 
and 77.9%, respectively. (Results are not shown in tables.) 

Reliability 

Internal consistency reliability of the appraisal form by 
Cronbach's a was 0.957 and that for availability, depend- 
ability and capability were 0.922, 0.961 and 0.937, respect- 
ively. Test-retest reliability of the appraisal form by Kappa 
is 0.976, and for the three dimensions were 0.862, 0.857, 
0.885, respectively. Inter-scorer reliability of the appraisal 
form, intra-class correlation coefficient verified test-retest 
reliability; ICC of the appraisal form was 0.976, and for the 
three dimensions, respectively, 0.862, 0.857, 0.885. 

Discussion 

In this 15-month study, we designed, developed and 
tested an appraisal form for assessing the effectiveness of 
adult invasive mechanical ventilation systems, the EAP- 
AIMVS. Our design was based on the WSEIAC model 
of system effectiveness [17], which had been successfully 
applied to measure effectiveness of other types of sys- 
tems [18-21]. To our knowledge, this is the first appraisal 
form developed for mechanical ventilation systems using 
the WSEIAC model. Testing of the appraisal form with 
a convenience sample of 200 critical care patients on 
mechanical ventilation revealed that the three dimen- 
sions of effectiveness according to the WSEIAC model, 
namely system availability, dependability, and capability, 
had acceptable internal consistency (overall 0.957, and 
for the three dimensions, 0.922, 0.961 and 0.937, re- 
spectively), reliability, and test-retest reliability. As such, 
this indicates that the appraisal form can feasibly be ap- 
plied to the assessment of mechanical ventilation sys- 
tems by the responsible staff in clinical settings. 

Briefly, we must note possible differences in the inten- 
sive care clinical setting in China compared to those in 
other countries. To ensure providing optimum therapeutic 
requirements for critically ill patients, patients are placed 
together in a dedicated area in single units of about 15- 
square meters with necessary equipment (e.g., bedside and 
central monitors, multi-function breathing and electrocar- 
diograph machines, defibrillators, pacemakers, infusion 
pumps and equipment for tracheal intubation and trache- 
otomy). Nurses specially trained in basic knowledge and 
techniques of intensive care medicine are the primary care 
staff for critically ill patients. Medical and nursing staff 
members working independently are responsible for allo- 
cation of all equipment and nurses are directly responsible 
for airway care for patients requiring mechanical ventila- 
tion, including the system, all related interventions and 
weaning from the ventilator system. 
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Table 2 Re-arranged EAP-AIMVS form via Exploratory Factor analysis 



Item 



Eigenvalue Factor loading Variance explained by factor 



Factor 1 Availability 7 85 

1 . Check and evaluate function of ventilator 

2. Select available types and styles of endotracheal or tracheostomy tubes 

3. Nursing staff has the necessary knowledge and skills to manage machanical 
ventilation system 

4. Various policies and procedures related to mechanical ventilation are available 

5. Environment is suitable for ventilated patient 

Factor 2: Dependability 4.43 

6. Ensure that ventilation tubing is not twisted and that it is adequately supported 
so as not to pull on ETT/trachi 

7. Check placement of tube by listening for equal bilateral breath sounds 

8. Ensure that endotracheal tube or tracheostomy tube is held securely in position 
but not too tightly to result in pressure area lesions 

9. Stabilize the tube while turning or moving the patient 

10. When possible, elevate head of bed to 30°to45° to prevent ventilator-associated 
pneumonia 

1 1 . Maintain proper cuff pressure and check if necessary to prevent leakage of air 
and contaminated secretions 

12. Suction oropharyngeal and tracheal secretions, more often if necessary 
to maintain a patent airway 

13. Oral care at least once a shift and more often if indicated 

14. Carry out appropriate airway humidification to prevent sticky sputum 
and keep patient comfortable 

15. Ventilator circuits are changed weekly or as necessary 

16. HME filters and end expiratory filters are changed routinely every 24 hours or 
more frequently if condensation is visible 

17. Assess patients' psychological state and help them develop individualized 
nurse-patient communication method and plan 

Factor 3: Capability 4 .22 

18. Tube displacement 

19. Unplanned extubation 

20. Airway obstruction 

21. Ventilator-associated pneumonia 

22. Adverse psychological reactions 
Total EAP-AIMVS 



0.72 



35.6 



20.15 



19.18 



75.01 



Results of eigenvalues, factor loadings, and variance explained by exploratory factor analysis with varimax rotation. 



The appraisal form has a solid theoretical basis 

The design and development of a research tool requires a 
powerful theoretical basis [45]. The appraisal form for ef- 
fectiveness of mechanical ventilation systems as advanced 
in this research, and which relied on the WSEIAC model 
as a basic framework complemented by integrated litera- 
ture review and expert evaluation, combined the three 
dimensions of availability, dependability and capability to 
evaluate essential connections and related factors of 
mechanical ventilation systems, including pre-use state, 
functioning procedure and results, reflecting the quality 
control of nursing from pre-use until end-stage. The ap- 
praisal form is intended to assist clinical nurses in 



recognizing key indicators of mechanical ventilation ef- 
fectiveness. Effectiveness of nursing intervention for the 
whole procedure or different stages can be assessed by 
using either the whole appraisal form or its subscales, as 
suggested by Noar (2003) [47]. 

The appraisal form has good construct validity 

As evaluated in the second part of this study, the appraisal 
form exhibited good construct validity. The three com- 
mon factors, corresponding to the three dimensions of 
system effectiveness in the WSEIAC model, were ex- 
tracted via exploratory factor analysis, and its accumulated 
variance contribution rate was 75.01%. Factor classification 
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Figure 1 This figure presents the measurement model evaluated outside the structural equation model (SEM) with correlation 
coefficients for the relationships among latent variables and standardized betas for the factor loadings. Availability, dependability, and 
capability are used as defined from EFA approach. Availability is presented by five indicators (item 1 to 5), dependability is presented by 12 
indicators (item 6 to 17), and capability is presented by five indicators (item 18 to 22) The final modified model yielded Chi-square = 34.79 
(p<.001), df=13,x2/df=2.68, GFI = 0.98, AGFI = 0.94, NFI = 0.95, PNFI = 0.44, PGFI = 0.35, RMSE = 0.07. 



was followed with the varimax orthogonal rotation of 
each item, which was even and clear, and the loading 
capacity of items with related factors was more than 0.4. 
This indicates good construct validity of the newly 
designed appraisal form. Limitations of estimating rela- 
tionships between exploratory factor analysis and vari- 
ables can indicate uncertainty about the relationships. 
Therefore, after using exploratory factor analysis as a 
base, this research then used confirmatory factor analysis 
to test the level of fitting of the three dimensions model 



structure with the structural equation model as a founda- 
tion. Results of confirmatory factor analysis proved that 
the three dimensions structure of the appraisal form is 
highly consistent with the WSEIAC model theoretical 
framework. This result also reinforced the construct val- 
idity of the appraisal form. 

The appraisal form has good congruent validity 

Congruent validity demonstrates how a scale differenti- 
ates characteristics of different groups of subjects [44]. 
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Table 3 Summary of distribution of EAP-AIMVS re-arranged after EFA for 200 subjects by ventilation status (N = 200) 



Item 



Total 

(N = 200) 



Weaning 
(N = 123) 



Die during 
ventilation(N = 77) 



Factor 1:Availability 

1 . Check ventilator function 3.63 ± 0.49 

2. Select available types and styles of endotracheal or tracheostomy tubes 3.6 ±0.49 

3. Nursing staff has the necessary knowledge and skills to manage 3.66 + 0.48 
mechanical ventilation system 

4. Various policies and procedures related to mechanical ventilation 3.62 + 0.49 
are available 

5. Environment is suitable for ventilated patient 3.59 + 0.52 
Factor 2: Dependability 

6. Ensure that ventilation tubing is not twisted and that it is 3.3 ± 0.69 
adequately supported so as not to pull on the ETT/trachi 

7. Check placement of tube by listening for equal bilateral breath sounds 3.32 + 0.69 

8. Ensure that endotracheal tube or tracheostomy tube is held securely in 3.1 ±0.65 
position but not too tightly to result in pressure area lesions 

9. Stabilize the tube while turning or moving the patient 3.1 2 ± 0.71 

10. When possible, elevate head of bed to 30° to 45° prevent 3.29 ±0.69 
ventilator-associated pneumonia 

1 1 . Maintain proper cuff pressure and check if necessary to prevent 3.26 ± 0.72 
leakage of air and contaminated secretions 

12. Suction oropharyngeal and tracheal secretions, more often if necessary, 3.34 ±0.68 
to maintain a patent airway 

1 3. Oral care at least once a shift and more often if indicated 3.3 ± 0.66 

14. Carry out appropriate airway humidification to prevent sticky sputum 3.28 ±0.69 
and keep patient comfortable 

1 5. Ventilator circuits are changed weekly or as necessary 3.3 ± 0.71 

16. HME filters and end expiratory filters are changed routinely every 24 hours 3.29 ±0.69 
or more frequently if condensation is visible 

17. Assess patients' psychological state and help them develop 3.28 ±0.7 
individualized nurse-patient communication method and plan 

Factor 3: Capability 

18. Tube displacement 3.1 6 ± U 

19. Unplanned extubation 3.4 ±0.9 

20. Airway obstruction 3.36 ± 0.93 

21 . Ventilator-associated pneumonia 3.2 ± 1 .03 

22. Adverse psychological reactions 3.1 1 ± 1 .08 
Total EAP-AIMVS 73.76 ± 1 1 .62 



3.84 + 0.37 
3.81 ±0.39 

3.85 ± 0.36 

3.81 ±0.39 
3.78 + 0.42 

3.6 ± 0.54 

3.63 + 0.52 
3.34 ± 0.54 

3.39 ± 0.6 
3.58 ±0.57 

3.58 ±0.57 

3.66 ±0.51 

3.61 ±0.51 

3.58 ±0.54 

3.61 ±0.55 

3.59 ±0.54 

3.58 ±0.57 



3.69 + 0.68 
3.8 ±0.55 
3.75 ±0.58 
3.63 + 0.69 
3.58 + 0.71 
80.41 ±6.29 



3.29 + 0.45 
3.25 + 0.43 
3.35 ± 0.48 

3.31 ±0.47 

3.29 ±0.53 

2.82 ± 0.62 

2.82 ± 0.64 
2.7 + 0.61 

2.68 ± 0.66 

2.82 + 0.6 

2.74 ± 0.62 

2.83 ± 0.62 

2.81 ±0.59 
2.79 ±0.61 

2.79 ± 0.64 
2.81 ±0.63 

2.81 ±0.63 



2.31 ±1.1 
2.74 + 0.95 
2.73 + 1.03 
2.51 ±1.11 
2.35 ±1.13 
63.1 3 ±10.19 



EFA, Exploratory Factor analysis. 

Data were summarized as mean ± Standard deviations. 



Esteban and Anzueto (2002) conducted an extensive, 
foresighted and multi-center research study, finding that 
patients' survival rate and prognosis were influenced by 
different factors of pre-use of mechanical ventilation, 
and complications and management of patients in mech- 
anical ventilation [47]. As an example of classical test 
theory, results of that study demonstrated that the as- 
sessment index, including management of mechanical 
ventilation, nursing management of airways and devel- 
opment of complications, influences the prognosis of 



patients [47]. Therefore, it is commonly understood that 
the poorer the prognosis of ventilation in a given pa- 
tient, the lower the effectiveness score would be. Con- 
gruent validity demonstrates whether or not the 
appraisal form reflects differences between factors. In 
the present study, results showed that patients weaned 
from the ventilator successfully had higher effectiveness 
scores than the patients who died during ventilation 
(t = 13.38,P < 0.05), indicating acceptable congruent val- 
idity of the appraisal form. 
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The appraisal form demonstrated good reliability 

Reliability refers to the level of credibility and stability of 
a scale and its evaluation results. Reliability can be eval- 
uated by various means. This study reviewed internal 
consistency reliability, test-retest reliability and inter- 
scorer reliability. Cronbach's a is an estimate of internal 
consistency and will typically increase as relationships 
between the test items increase. In the present study, 
Cronbach's a of the whole appraisal form and its differ- 
ent dimensions achieved the measurement model of 
more than 0.7, which indicates internal consistency reli- 
ability of the appraisal form, in keeping with previously 
published results [48]. Our result indicates good reliabil- 
ity of the effectiveness appraisal form and the homology 
of items that form each dimension. Test-retest reliability 
and inter-scorer reliability also achieved the measure- 
ment model of more than 0.75 [49]. It is shown that the 
variance of these two tests of the appraisal form is rela- 
tively low and stable, which indicates that the appraisal 
form is suitable for use by different clinical assessors. 

Prior to actually designing the EAP-AIMVS, we theo- 
rized that availability of a standardized form for assessing 
the effectiveness of adult invasive mechanical ventilation 
would be a valuable adjunct for nurses in Chinese hospi- 
tals caring for ventilated critical care patients, not only to 
help address nursing issues but to provide safer care for 
patients. This was the rationale for developing the EAP- 
AIMVS as a practical, theory-based form for appraising ef- 
fectiveness of mechanical ventilation systems, and during 
development we considered exactly how an effectiveness 
appraisal form might help to ensure patient safety and re- 
duce complications, morbidity and mortality. Other nurs- 
ing interventions reported in the literature provided 
encouragement. In a study documenting the need for edu- 
cation of nurses and other medical professionals who care 
for ventilated patients, the need for additional education 
and training regarding clinical practice in intensive care 
was expressed by 62% of medical staff, including intensi- 
vists and 42% of nurses [1]. Although having an effective- 
ness appraisal form will not take the place of education 
and training, the effectiveness evaluation can serve as a 
readily available guide or checklist that can be followed by 
ICU caregivers to help ensure safe, effective ventilation 
and efficient monitoring. Studies of weaning from mech- 
anical ventilation illustrate another aspect of standardizing 
care. Because prolonged mechanical ventilation is asso- 
ciated with high morbidity, especially ventilator associated 
pneumonia and lung injury, and increased mortality, 
patients receiving mechanical ventilation may actually 
spend 40% of the time being weaned from ventilation; pro- 
longed ventilation may also preclude patients' recovery 
from their primary critical illness [50] . The investigators in 
that study recommended surveying practitioners and 
multidisciplinary teams to collect the best ways to put 



optimal weaning methods into practice and then apply 
them [50]. Relative to weaning methods, routine applica- 
tion of a nurses' protocol-directed weaning procedure ap- 
plied in a French hospital resulted in improved outcomes 
and clinical benefits in patients requiring more than 48 
hours of mechanical ventilation [2]. In our study, routinely 
following a simple, step-by-step appraisal form for mech- 
anical ventilation effectiveness allowed more than 60% of 
patients to be weaned successfully, helped to prevent com- 
plications and reassuring staff that proper procedures had 
been carried out for system effectiveness, which implies 
patient safety. We made sure that effectiveness criteria 
included items such as ventilator circuits and secretion 
management, which have been shown to influence rates of 
ventilator-associated pneumonia [13]. We also included 
the intensive care physical environment as a factor in the 
appraisal (Item 5, Table 3) since a healthy environment is 
needed to protect against airborne infection and prevent 
cross-infection; hand washing and aseptic technique must 
be reinforced, air quality must be monitored, visitors 
restricted and ventilation equipment needs disinfecting on 
a regular basis. Also, a 28-day international study of venti- 
lated patient characteristics and outcomes suggested that 
survival was associated not only with baseline factors 
present at the inception of mechanical ventilation but the 
development of complications and the management of 
patients [47] . Thus, future research on mechanical ventila- 
tion effectiveness must evaluate patient data, including 
complications, unexpected developments and outcomes, at 
baseline and during the course of mechanical ventilation. 

Implications for practice 

Results of the present study have provided an evaluation 
form by which nursing staff or other responsible staff 
such as respiratory therapists can evaluate the effective- 
ness of mechanical ventilation systems and related care 
aspects of nursing interventions for mechanical ventila- 
tion patients. Specifically, the developed form can help 
clinical nursing staff to differentiate key indexes and in- 
crease work efficiency. Future research should focus on 
progressive evaluation of mechanical ventilation effi- 
ciency in China and abroad, and collect clinical practice 
data for ventilation technique and management of artifi- 
cial airways. Guiding principles for evaluation of mech- 
nical ventilation systems can be improved and 
progressive development of appraisal forms may ultim- 
ately include software applications that evaluate each 
item according to individual patients' clinical situation. 

Limitations 

First, the appraisal form developed in this study for 
assessing the effectiveness of mechanical ventilation was 
not developed on the basis of an existing scale but was 
an initial attempt to develop an original appraisal form 
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and evaluate its suitability as an assessment tool. The 
sample was also relatively small. Assessment items of 
the appraisal form therefore require further evaluation 
with a larger clinical sample. Secondly, convenience 
sampling method was used and its representativeness is 
limited. Sampling range and quantity should be broa- 
dened in future research, and the appraisal form will be 
further developed and improved through clinical evi- 
dence and progressive mechanical ventilation research. 

Conclusions 

The theory-based appraisal instrument to assess nursing 
interventions for mechanical ventilation systems demon- 
strates acceptable internal consistency, inter-scorer reli- 
ability, and test-retest reliability and can reasonably 
guide clinical nursing staff to evaluate essential monitor- 
ing techniques and factors related to mechanical ventila- 
tion. The appraisal form is appropriate for use in the 
clinical setting and its application may help to improve 
the effectiveness and performance of adult mechanical 
ventilation systems as well as helping to provide safe, ef- 
fective treatment for patients under intensive care. 

Additional file 
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